System and method for imaging reflecting objects

ABSTRACT

A method and system for automated visual inspection of an object include using different patterns of illumination, each pattern including constant frequency pulses of illumination. The pulses of each different pattern of illumination are temporally offset so as to enable obtaining images illuminated by one or another illumination pattern. The images are combined using different parts of the different images in which the most details of the object are available, to produce a full image of the object which is essentially glare-free. The use of constant frequency pulses enables obtaining different pattern illumination images, to enable creating a glare-free image of an object, while providing a flicker-free inspection environment for human workers.

FIELD

The present invention relates to visual inspection processes, forexample, inspection of items on a production line.

BACKGROUND

Inspection during production processes helps control the quality ofproducts by identifying defects and acting upon their detection, forexample, by fixing them or discarding the defected part, and is thususeful in improving productivity, reducing defect rates, and reducingre-work and waste.

Automated visual inspection methods are used in production lines toidentify visually detectable anomalies that may have a functional oresthetical impact on the integrity of a manufactured part. Existingvisual inspection solutions for production lines, using cameras, rely oncustom made automated visual inspection systems, which are typicallyhighly expensive and require expert integration of hardware and softwarecomponents, as well as expert maintenance of these in the life-time ofthe inspection solution and the production line.

When using automated visual inspection, image quality affects theability of a processor running algorithms for inspection, to reliablycarry out inspection tasks, such as, defect detection, quality assurance(QA), sorting and/or counting, gating, etc.

For instance, glossy or reflective items (such as pills or other itemsin a clear plastic wrapping) usually have a surface that reflects lightin a specular (mirror-like) direction, as opposed to matte objects thatreflect light diffusely, in many directions. Other factors that canaffect gloss include the refractive index of the material, the angle ofincident light and the surface topography. Due to the specularreflection of glossy objects, images of glossy objects will typicallyinclude a glare, which can obscure details of the imaged object. Thus,images of glossy objects are unsuitable for visual inspection tasks.

SUMMARY

Embodiments of the invention provide a system and method for obtaining asubstantially glare-free image of an item, during visual inspectionprocesses, for any type of item, with minimal setup.

Different illumination and possibly different exposure patterns may beautomatically tested during a setup stage of the inspection process, todetermine the illumination/exposure pattern(s) that will enablemaximizing the information collected for each object and enableobtaining a substantially glare-free image for each object type.

The illumination/exposure pattern determined during the setup stage canthen be used to obtain images of same-type objects during the inspectionstage. Additionally, illumination patterns may be automatically adjustedbased on orientation of the object in the image, such that even ifobjects are orientated in the inspection stage differently than in thesetup stage, the information collected during the setup stage can stillbe used to determine which illumination patterns to use during theinspection stage.

Testing different illumination/exposure patterns during the setup stageenables to determine the minimal set of different illumination/exposurepatterns required for obtaining maximal information of the object.Because adding each illumination/exposure pattern requires capturinganother image, which prolongs the overall inspection time, determiningthe minimal number of required illumination/exposure, patterns providesa shorter overall inspection time.

A system for automated visual inspection, according to embodiments ofthe invention, includes a camera configured to capture images of anobject on an inspection line, a light source to illuminate at least aportion of the camera field of view (FOV), and a processor incommunication with the camera and light source.

In some embodiments the light source produces light pulses. Theprocessor controls the light source to differentially illuminate thecamera FOV and ensures that camera exposure events are insynchronization with the light pulses, to produce a plurality ofdifferent images, each being captured in a different illuminationpattern.

In one embodiment the processor controls one segment of the light sourceto illuminate in a first pattern of pulses and a second segment toilluminate in a second pattern of pulses. Typically, the pulses of thefirst pattern and second pattern are each at a constant frequency. Thepulses of the first pattern and second pattern may be offset from eachother, such that there are times of overlapping pulses and times of nooverlap. The processor may control an exposure event of the camera tocoincide with a time in which there is no overlap of pulses between thefirst pattern and second pattern. An image captured during this exposuretime may be used to detect the object in the image and/or may be used asone of a plurality of different illumination pattern images combined toprovide a combined image for visual inspection.

In some embodiments, the processor may control a first exposure event ofthe camera to coincide with a time in which pulses in both the firstpattern and second pattern overlap and a second exposure event tocoincide with a time in which pulses in the first pattern and secondpattern do not overlap. The image captured during the first exposureevent may be used to detect the object in the image and/or may be usedas one of the different illumination pattern images combined to providea combined image for visual inspection.

The processor may then determine which of the plurality of images, whencombined, provide a combined image of the object showing the most detailof the object. The determined images may then be combined to provide acombined image, which is a substantially glare-free image. Visualinspection can be performed on the combined image.

The term “visual inspection” may include displaying the image to a userand/or applying visual inspection algorithms on the image. As furtherdetailed herein, a “visual inspection algorithm” refers to a sequence ofautomatically performed steps that are designed to detect objects on aninspection line, from images, and classify the objects based onrequirements of the inspection process. For example, a requirement of aninspection process may be to detect defects on the object and/or performother inspection tasks, such as QA, sorting and/or counting, gating,etc. Visual inspection algorithms, according to embodiments of theinvention, typically include using computer vision techniques.

BRIEF DESCRIPTION OF THE FIGURES

The invention will now be described in relation to certain examples andembodiments with reference to the following illustrative figures so thatit may be more fully understood. In the drawings:

FIGS. 1A and 1B schematically illustrate a setup stage and an inspectionstage according to embodiments of the invention;

FIG. 1C schematically illustrates a system for automated visualinspection, according to embodiments of the invention;

FIGS. 2A and 2B schematically illustrate different illumination patternsand camera exposure timing, according to embodiments of the invention;

FIG. 3 schematically illustrates a method for obtaining a substantiallyglare-free image for visual inspection, according to embodiments of theinvention;

FIG. 4 schematically illustrates a method for determining which of theplurality of images provide a combined image of the object showing themost detail of the object, according to embodiments of the invention;

FIGS. 5A and 5B schematically illustrate methods for obtainingsubstantially glare-free images for visual inspection in an inspectionstage, based on processing in the setup stage, according to embodimentsof the invention;

FIG. 6 schematically illustrates a method for obtaining a substantiallyglare-free image for visual inspection, according to another embodimentof the invention; and

FIG. 7 schematically illustrates a timeline of a visual inspectionprocess, including obtaining a glare-free HDR image, according to anembodiment of the invention.

DETAILED DESCRIPTION

In the following description, various aspects of the present inventionwill be described. For purposes of explanation, specific configurationsand details are set forth in order to provide a thorough understandingof the present invention. However, it will also be apparent to oneskilled in the art that the present invention may be practiced withoutthe specific details presented herein. Furthermore, well known featuresmay be omitted or simplified in order not to obscure the presentinvention.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions utilizing terms such as “analyzing”, “processing,”“computing,” “calculating,” “determining,” “detecting”, “identifying”,“creating”, “producing”, “obtaining”, “applying” or the like, refer tothe action and/or processes of a computer or computing system, orsimilar electronic computing device, that manipulates and/or transformsdata represented as physical, such as electronic, quantities within thecomputing system's registers and/or memories into other data similarlyrepresented as physical quantities within the computing system'smemories, registers or other such information storage, transmission ordisplay devices. Unless otherwise stated, these terms refer to automaticaction of a processor, independent of and without any actions of a humanoperator.

The terms “item” and “object” may be used interchangeably and are meantto describe the same thing.

The terms “same-type items” or “same-type objects” refer to items orobjects which are of the same physical makeup and are similar to eachother in shape and dimensions and possibly color and other physicalfeatures. Typically, items of a single production batch or series, orbatch of items in the same stage on the production line, may be“same-type items”. For example, if the inspected items are sanitaryproducts, different sink bowls of the same batch are same-type items.Same type items may differ from each other within permitted tolerances.

A defect may include, for example, a visible flaw on the surface of theitem, an undesirable size of the item or part of the item, anundesirable shape or color of the item or part of the item, anundesirable number of parts of the item, a wrong or missing assembly ofinterfaces of the item, a broken or burned part, and an incorrectalignment of the item or parts of the item, a wrong or defected barcode,and in general, any difference between the defect-free sample and theinspected item, which would be evident from the images to a user,namely, a human inspector. In some embodiments a defect may includeflaws which are visible only in enlarged or high-resolution images,e.g., images obtained by microscopes or other specialized cameras.

Typically, a visual inspection process uses images of items confirmed bya user, as references to which unconfirmed images of same-type items arecompared, to detect defects on the item in the unconfirmed image or forother inspection tasks, such as QA, sorting, gating, counting and more.The user confirmed images (also referred to as “reference images”) areusually obtained during a setup stage prior to an inspection stage.

FIGS. 1A and 1B schematically illustrate a setup stage and inspectionstage, correspondingly, according to embodiments of the invention.

In the setup stage, two or more samples of a manufactured item of thesame type, (in some embodiment, the samples are items with no defects),e.g., defect free sample items 2 and 2′, are placed in succession withina field of view (FOV) 3′ of (one or more) camera 3. For example, defectfree sample items 2 and 2′ may be placed on an inspection line whichincludes conveyor belt 9 such that movement of the conveyor belt 9 firstbrings item 2 into the FOV 3′ and then brings item 2′ into the FOV 3′.Images captured by camera 3 may be displayed on a user interface device6.

Each defect free sample item 2 and 2′ is illuminated by light source 5and imaged by camera 3. These images, which may be referred to as setupimages or reference images, may be obtained under different conditions,as described below, for example, by using in each frame differentillumination patterns and/or different imaging parameters of camera 3,for example different focuses and exposure times. A processor mayanalyze the relation between different images of a same type of itemwhich were obtained under the different conditions. This analysis duringthe setup stage enables to continually optimize the illumination andimaging parameters with minimal processing time during the followinginspection stage.

In one embodiment, the analysis of the setup images is used to determinea spatial range in which the items (e.g., items 2 and 2′) show nosignificant perspective distortion when aligned with another same typeof item. The level of perspective distortion between items in differentimages can be analyzed, for example, by detecting regions in an itemwhich do not have corresponding features between the setup images, byanalyzing the intersection location and angles between the item'sborders or marked areas of interest on the item, etc. The borders of thespatial range may be calculated by comparing two (or more) setup images(in which items may be positioned and/or oriented differently) anddetermining which of the images show perspective distortion and which donot.

The calculated range can then be used to determine the borders of whereand/or in which orientation, scale or other dispositioning, an inspecteditem may be placed on the inspection line so as to avoid distortion whencompared with the reference images. Additionally, by using a set ofsetup images as references for each other, the processor can detectimages having similar spatial decomposition and this set of images canthen be analyzed to see if there are enough similar setup images toallow registration, defect-detection and other analyses for eachpossible positioning of the item on the inspection line.

Analysis of the setup images may be performed to collect characteristicsof an item, information regarding possible 2D shapes and 3Dcharacteristics (e.g., rotations on the inspection line) of an item orto find uniquely discriminative features of the item and the spatialrelation between these unique features, as preserved between the setupimages. Also, complete representation of a type of item is achieved whenthe range of shape tolerance characteristic to this item and the surfacevariations characteristic to this item are all manifested in the set ofsetup images.

Based on the information collected from setup images, a processor candetect a second item of the same type and perform inspection tasks, evenif the second item was not previously learned by the processor. Thisallows the processor to detect when a new item (of the same type) isimaged, and then to analyze the new item, for example, to run visualinspection algorithms that typically include comparing images of the newitem to reference images of the same type item to identify differencesbetween the reference and new item images, e.g., to search for a defecton an inspected item. Other tasks performed by the visual inspectionalgorithms may include sorting between different objects on theproduction-line, decoding codes (such as DMC, QR codes and others),counting the number of objects currently on the production-line, andothers.

Instructions to a user regarding adjustment of camera and/orillumination parameters can be displayed to the user via a userinterface device 6. Once it is determined, based on the analysis of thereference images, that enough information about the item is obtained,the setup stage may be concluded and a notification is displayed orotherwise presented to a user, via user interface device 6, to stopplacing samples (sample items 2 and 2′) on the conveyor belt 9 and/or toplace on the conveyor belt 9 inspected items 4, 4′ and 4″ (as shown inFIG. 1B).

In the inspection stage (which is schematically illustrated in FIG. 1B)that follows an initial setup stage, inspected items 4, 4′ and 4″, whichare of the same type as sample items 2 and 2′ and which may or may nothave defects, are imaged in succession by camera 3. These images, whichmay be referred to as inspection images, are analyzed using visualinspection algorithms, which include using computer vision techniques(e.g., including machine learning processes) to enable performinginspection tasks (such as, defect detection, QA, sorting and/orcounting) on items 4, 4′ and 4″. In the example illustrated in FIG. 1B,item 4′ includes a defect 7, whereas items 4 and 4″ are defect free. Inembodiments of the invention the inspection images are illuminated bylight source 5 in a pattern which has been determined during the setupstage.

Setup processes may be performed prior to the inspection stage andduring the inspection stage. In some embodiments, reference images maybe analyzed (e.g., as described above) throughout the inspectionprocess, not necessarily only during an initial setup stage, an exampleof which is described in FIG. 1A. For example, an inspection imagelabeled by a user and/or by the visual inspection algorithm (e.g.,either as defected or defect-free) during the inspection stage, may thenbe saved as a new reference image (e.g., in a reference image database,as described below) to be analyzed and possibly update the informationcollected during the initial setup stage.

Although a particular example of a setup and inspection stage of avisual inspection process is described herein, it should be appreciatedthat embodiments of the invention may be practiced with other setup andinspection procedures of visual inspection processes.

An exemplary system which may be used for automated visual inspection ofan item on an inspection line, according to embodiments of theinvention, is schematically illustrated in FIG. 1C. In one embodimentthe system includes a processor 102 in communication with one or morecamera(s) 103 and with one or more light source(s) 105. Processor 102may also be in communication with a device, such as a user interfacedevice 106 and/or other devices, such as storage device 108.

Components of the system may be in wired or wireless communication andmay include suitable ports and/or network hubs. In some embodimentsprocessor 102 may communicate with a device, such as storage device 108and/or user interface device 106 via a controller, such as aprogrammable logic controller (PLC), typically used in manufacturingprocesses, e.g., for data handling, storage and processing power andcommunication capabilities. A controller may be in communication withprocessor 102, storage device 108, user interface device 106 and/orother components of the system (such as camera 103 and light source105), via USB, Ethernet, appropriate cabling, etc.

Processor 102 may include, for example, one or more processors and maybe a central processing unit (CPU), a graphics processing unit (GPU), adigital signal processor (DSP), a field-programmable gate array (FPGA),a microprocessor, a controller, a chip, a microchip, an integratedcircuit (IC), or any other suitable multi-purpose or specific processoror controller. Processor 102 may be locally embedded or remote, e.g.,cloud based.

The user interface device 106 may include a display, such as a monitoror screen, for displaying images, instructions and/or notifications to auser (e.g., via text or other content displayed on the monitor). Userinterface device 106 may also be designed to receive input from a user.For example, user interface device 106 may include a monitor andkeyboard and/or mouse and/or touch screen, to enable a user to inputfeedback.

Storage device 108 may be a server including for example, volatileand/or non-volatile storage media, such as a hard disk drive (HDD) orsolid-state drive (SSD). Storage device 108 may be connected locally orremotely, e.g., in the cloud. In some embodiments, storage device 108may include software to receive and manage image data related toreference images. A reference image database may be located at storagedevice 108 or at another location.

Camera(s) 103, which are configured to obtain an image of an object 130on an inspection line 109, are typically placed and possibly fixed inrelation to the inspection line 109 (which may include e.g., a conveyerbelt, a robotic arm, etc.), such that items placed on the inspectionline are within the FOV 103′ of the camera 103.

Typically, inspection line 109 moves, e.g., in direction of arrow 19,thereby moving the items on it. Each item 130 is within the field ofview 103′ of the camera 103 for a certain amount of time, termed hereinan “inspection window”, and is then moved out of the camera FOV 103′. Inother embodiments, the inspection line need not move, rather the camera103 may be moved to capture each of items 130 within its field of view103′.

An inspection line typically operates to repetitively run inspectionwindows. An inspection window may last several seconds, which means,depending on the frame capture rate of the camera 103, that severalimages of each item 130 may be captured in each inspection window. Inone embodiment the camera 103 captures images at a rate of 30 frames persecond (fps) or below, e.g., 20 fps, to obtain a video. In otherembodiments camera 103 operates at a frame rate of above 30 fps, forexample, in some embodiments, the camera 103 operates at 60 fps orabove.

Camera 103 may include a CCD or CMOS or another appropriate imagesensor. The camera 103 may be a 2D or 3D camera. In some embodiments,the camera 103 may include a standard camera provided, for example, withmobile devices such as smart-phones or tablets. In other embodiments thecamera 103 is a specialized camera, e.g., a camera for obtaining highresolution images. In some embodiments camera 103 may be designed toimage at IR or near IR wavelengths. For example, the camera 103 mayinclude a suitable filter.

The system also includes a light source 105, to illuminate at least aportion of the camera 103 field of view 103′. In one embodiment (whichis schematically shown in FIG. 1C), light source 105 surrounds camera103. Light source 105 may include segments, each segment being capableof illuminating independently from other segments and each segmentcapable of being controlled independently from the other segments. Forexample, light source 105 may include separate LEDs or groups of LEDs,which can be turned ON/OFF independently. The different segments oflight source 105 may be physically separated, e.g., by an opaque barrierplaced in between the segments.

In some cases, e.g., when using light sources that have a transformerthat causes a “switch-on delay” (e.g., LEDs), in order to avoid thedelay, turning the light source 105 ON/OFF includes increasing the lightintensity of the light source when “turning on” and greatly lowering theintensity (to a point where the light is negligible) when “turning off”,rather than fully powering off the light source.

One or more of each separate segment may include a diffuser (e.g., afilm of translucent material) to provide diffuse, uniform illumination.In one embodiment light source 105 includes LEDs of differentwavelengths, e.g., some of the LEDs may illuminate at near IR and someof the LEDs may illuminate white light. Turning each separate LED ON/OFFwill cause light source 105 to illuminate at a different wavelength. Inother embodiments, each segment (e.g., each LED or each group of LEDs)can be controlled to illuminate at a different intensity.

Processor 102 receives image data (which may include data such as pixelvalues that represent the intensity of reflected light as well aspartial or full images or videos) of objects on the inspection line(which are illuminated by light source 105) from the one or morecamera(s) 103, and runs processes according to embodiments of theinvention.

Processor 102 is typically in communication with one or more memoryunit(s) 112. Memory unit 112 may store at least part of the image datareceived from camera(s) 103.

Memory unit 112 may include, for example, a random access memory (RAM),a dynamic RAM (DRAM), a flash memory, a volatile memory, a non-volatilememory, a cache memory, a buffer, a short term memory unit, a long termmemory unit, or other suitable memory units or storage units.

In some embodiments the memory unit 112 stores executable instructionsthat, when executed by processor 102, facilitate performance ofoperations of processor 102, as described herein.

In one embodiment, processor 102 is in communication with the camera 103and light source 105, and controls the light source 105 to illuminatedifferent portions of the FOV 103′ in synchronization with camera 103exposure events. This way, a plurality of different-illumination-patternimages of object 130 are obtained. In eachdifferent-illumination-pattern image, different areas of the object maybe differently illuminated.

For example, as schematically illustrated in FIG. 2A, light source 105may include six different segments A, B, C, D, E and F. For example,light source 105 may include a flat dome light (which includes adiffusor and a hole template on the diffusor) with six differentlycontrolled segments. Alternatively, light source 105 may include six (oranother number) surrounding spot lights, each spot light illuminating aportion of the camera FOV 103′, and all six spot lights togetherilluminating the whole FOV 103′ of the camera 103. Other numbers ofsegments and segmentation options of the light source can be usedaccording to embodiments of the invention. For example, four differentilluminating segments may surround a camera and may be differentlycontrolled to provide different-illumination-pattern images.

In one example, processor 102 may simultaneously control one segment toilluminate and another segment to be shut off. In other examples,processor 102 can control the different segments to simultaneouslyilluminate at different intensities. For example, one or a few segmentscan be controlled to illuminate at a high intensity and another segment(or segments) can be controlled to illuminate at a low intensity.

In one example, pulse duration modulation (PDM) may be used to providedifferent intensities from light source 105, as further exemplifiedbelow.

Processor 102 controls light source 105 to illuminate differentillumination patterns, typically in synchronization with exposure eventsof the camera 103.

Light source 105 may illuminate high-frequency pulses of light to enableobtaining several short exposure images of each object 130, allowing ashorter overall imaging time, which among other benefits, allows fastersampling of a moving object while it is still within a single inspectionwindow. In some embodiments, some of the light pulses are at a highintensity, to enable capturing well-lit images of the item 130, whereasthe other pulses are at a lower intensity, so as to prevent quick burnout of the light source. The high intensity pulses may be specificallytimed (or the camera exposure events can be specifically timed) toenable capturing several images of the object while it is still withinthe inspection window and before the object has moved (e.g., due tomovement of the inspection line) too much, to enable capturing images ofthe object from the same point of view and without blurring due to themotion effect.

Using high frequency pulses of high intensity light in combination witha camera operating at a high frame rate (e.g., above 30 fps, such as, 60fps or above), enables capturing several images of the object (eachimage with a different illumination pattern) within a short period oftime, thereby reducing the issues created by imaging moving objects(such as blurriness and changing points of view, as discussed above).

FIG. 2A illustrates an exemplary differential illumination schedule. Ina first phase I, camera 103 captures images (e.g., image 21) in videomode and all segments of light source 105 are lit, typically at lowintensity (which enables energy saving and will not exceed the powerconsumption supported by the light source hardware). For example, lightsource 105 may include six 12 W LEDs. The 12 W LEDs can be operated at a50% duty cycle (i.e., can be turned on 50% percent of the time and off(or very low) 50% of the time) during phase I to provide medium or lowintensity illumination. Object 130 is visible in image 21 however notall details of the object are clear (as indicated by the dashed lines),due to the illumination being reflected off the object and/or because oflow intensity of the illumination.

Once object 130 is detected in the low intensity illumination image 21(and possibly determined to be in a predetermined spatial range on theinspection line, as discussed above) processor 102 controls light source105 to transition to phase II, in which all segments of light source 105are lit at high intensity to enable getting a well-lit image 22 of theobject. For example, all six 12 W LED may be operated at 95% duty cycle(i.e., on 95% percent of the time and off (or very low) 5% of the time)during phase II. However, if object 130 has reflecting surfaces (if, forexample, object 130 is a coated PCB or plastic or glass object) image 22may show reflection (glare), thereby obscuring some of the details ofobject 130. In this case, processor 102 controls light source 105 toilluminate in several different partial patterns, such as in phasesIII-VI.

In phase III only segment B is on, e.g., at 50% or higher duty cycle,and segments A, C, D, E and F are off (or very low). Alternatively,segment B may be illuminating high intensity light, e.g., at 95% dutycycle, while one or more of segments A, C, D, E and F are illuminatinglow intensity light, e.g., at 50% duty cycle or lower, so as to avoidglare in certain parts of the image.

Similarly, in phase IV segment D is on and segments A, B, C, E and F areoff. Alternatively, segment D may be illuminating high intensity light,while one or more of segments A, B, C, E and F are illuminating lowintensity light. Similarly, in phase V segment F is on and segments A,B, C, D and E are off. Alternatively, segment F may be illuminating highintensity light, while one or more of segments A, B, C, D and E areilluminating low intensity light.

In phase VI segments A and C are on whereas the other segments are off.In phase VI, each of segment A and C may be illuminating at a differentintensity and/or different wavelength.

In each phase the segments that are on may be turned on in typicallyhigh frequency, short illumination pulses.

The illumination pulses are typically synchronized with the camera 103shutter such that, in the case exemplified in FIG. 2A, each ofdifferent-illumination-pattern images 21, 22, 23, 24, 25 and 26 isobtained during an exposure event of camera 103. In other embodiments,e.g., as described in FIG. 2B, different illumination patterns may atleast partially overlap in time, such that exposure events of the camera103 may capture several different illumination patterns simultaneously.

In each of images 21-26, different details of the object 130 are visiblewhile other details are obscured due to glare from different portions ofthe images. If images 21-26 were to be combined, each image“contributing” its visible details, the combined image would be awell-lit image of object 130 with all or a maximum (namely, a sufficientamount or most of) of its details visible to enable an inspection task,such as defect detection.

In order to avoid visible flickering in the video captured by camera 103and/or flickering that may be irritating for plant workers, highfrequency pulses of light may be used. For example, processor 102 maycontrol light source 105 to illuminate pulses of light at a frequencyhigher than the sampling frequency of a human eye, as pulses at afrequency higher than the sampling frequency of the eye will typicallynot be noticed by a person. Additionally, as long as illumination pulsesare at a consistent frequency, flickering (e.g., due to changes inillumination intensity) will not be noted by a person.

In one embodiment, processor 102 controls a first segment of lightsource 105 to illuminate in a first pattern of pulses and a secondsegment of light source 105 to illuminate in a second pattern of pulses.The pulses in each illumination pattern are at a constant, unvaryingfrequency.

In one embodiment, the pulses of the first pattern and second patternare offset from each other. In this embodiment, processor 102 maycontrol a first exposure event of camera 103 to coincide with a time inwhich a pulse in both the first pattern and second pattern overlap and asecond exposure event of camera 103 to coincide with a time of a pulsein either the first pattern or the second pattern, but in which there isno overlap of pulses in the first pattern and second pattern.

An image of an object 130 captured during the first exposure event canbe used to detect the object 130, whereas parts of images of object 130captured during the exposure event coinciding with a time of a pulse ofeither the first pattern or the second pattern (when there is no overlapof pulses), may be used (possibly together with at least a part of theimage captured during the first exposure event) to provide a combinedimage on which to apply inspection algorithms to provide inspection ofobject 130.

In one embodiment, which is schematically illustrated in FIG. 2B,illumination patterns and camera exposure events are synchronized suchthat an image illuminated by all light segments (e.g., image 21 or 22)as well as partially illuminated images (e.g., images 23, 24, 25 or 26)may be captured while maintaining a consistent frequency of light pulsesso as to provide a flicker-free inspection environment for humanworkers. For example, a first pattern of illumination (1) includes onlythe right-side segments being on (e.g., segments A, C and E) whereas asecond pattern of illumination (2) includes only the left side segmentsbeing on (e.g., segments B, D and F). Each of the patterns includeslight pulses (Pu) repeated at a constant, unvarying frequency, however,the pulses of the first pattern of illumination are temporally offset inrelation to the pulses of the second pattern of illumination. Thisoffsetting of pulses enables one camera exposure event (E1) to capturean image illuminated simultaneously by part of a light pulse (Pu1) fromthe first pattern and part of a light pulse (Pu2) from the secondpattern, thus obtaining an image illuminated by all segments (both rightside and left side segments).

At the time of a second camera exposure event (E2) the illumination inthe first pattern is on but the illumination in the second pattern isoff Thus, the image captured during E2 is illuminated by only a pulse(or part of the pulse) from the first pattern, namely, illuminated bythe segments on the right side. At the time of a third camera exposureevent (E3) the illumination in the first pattern is off but theillumination in the second pattern is on. Thus, the image capturedduring E3 is illuminated by only a pulse (or part of the pulse) from thesecond pattern, namely, illuminated by the segments on the left side.

The image captured during exposure event E1 may be used to detect anobject (e.g., object 130) on an inspection line. Images captured duringexposure events E2 and E3 provide different illumination pattern imagesthat may be combined (possibly together with parts of the image obtainedduring E1) to create a maximum-detail image of the object.

In some cases, a minimal number of images per object may be desired,e.g., to avoid blurriness and changing points of view when inspecting amoving object. In such embodiments, the camera may be set for lessexposure events, e.g., only events E2 and E3. An image captured duringexposure event E2 may be used to detect the object and to provide afirst illumination pattern image whereas the image captured duringexposure event E3 provides a second illumination pattern image. Thus,less images of the object may be used to create a maximum-detail imageof the object.

As schematically illustrated in FIG. 3 , processor 102 receives, in step302, a plurality of different-illumination-pattern images of an object(e.g., images 21-26 of object 130) and determines which of the pluralityof images are images having a specific illumination pattern, such thatwhen combined, provide a combined image of the object showing the mostdetail of the object (step 304). This step can be performed by comparingthe images or pixels of the images, to each other. In some embodiments,processor 102 will search for the minimal number of images required toachieve a combined image. Typically, a measure of information can bedetermined for each of the images and may be used in combining images,as further detailed below.

Processor 102 then combines the determined images to provide a combinedimage (step 306) and uses computer vision techniques on the combinedimage for visual inspection of the object 103 (step 308) and to enableperforming inspection tasks, such as, defect detection, QA, sortingand/or counting. In some embodiments, processor 102 causes the combinedimage to be displayed on user interface device 106.

In one embodiment, the method includes detecting the object in a firstimage of the plurality of images obtained in step 302, e.g., in image 21or 22. The object may then be detected in the combined image usingcharacteristics of the object detected in the first image, such as byusing spatial parameters of the object detected in the first image.

The object may be a whole object and/or a region of interest on anobject. In some embodiments a region of interest (ROI) may beautomatically detected by a processor, e.g., by using image analysistechniques. Pixels associated with a detected object (or ROI) may bedetermined by using image analysis algorithms such as segmentation. Insome embodiments, a processor receives indications of an outline (e.g.,borders) of the object from a user and determines which pixels areassociated with the object, possibly using segmentation and based on theborders of the object.

For example, based on user input of an ROI on an image of an item on aninspection line, a processor may create an outline or other indicationsurrounding the ROI. An indication of ROI, input by a user, may includecoordinates and/or may include a line, e.g., a colored line, a brokenline or other style of line, or polygon or other shape surrounding theregion of interest.

The ROI may be an area on the object which is associated with defectdetection. For example, an ROI may be an area on the object in which auser requires defect detection or an area on the object in which theuser does not require defect detection. Thus, visual inspectionalgorithms (e.g., processes to detect defects on items) may beconditionally applied, based on an ROI. Additionally, same-type objectsmay have permitted differences, which are not defects. For example,objects may have texture, pattern or color differences or moving partson the object surface, which are not considered to be defects. In someembodiments, these areas of permitted differences may be defined as ROIsin which visual inspection algorithms are not applied, thus avoidingfalse detection of defects.

In some embodiments, specific, limited areas may be defined in an image,which are ROIs in which glare cannot be tolerated. Processor 102 maycontrol light source 105 to differentially illuminate the image based ondetermination of an ROI. For example, if an ROI is determined on abottom right corner of an object (in an area covered by segments D andF, for example) the relevant segments of light source 105, that canprovide an image without glare at the bottom right corner of the object,may be turned on while other segments of light source 105 which createglare in the ROI may not be turned on.

In some embodiments, based on detection of an object during the setupstage, a same-type object can be easily detected in an image in theinspection stage, prior to, or in parallel to, obtainingdifferent-illumination-pattern images of the object. Additionally, theobject may be easily detected in a combined image based on detection ofthe object in at least one of the images used to create the combinedimage. For example, spatial properties and uniquely representingfeatures or attributes of the object may be detected in a first image ofthe object and may then be available when performing visual inspectionof a same-type object in a combined image, thereby saving time byavoiding the need to detect these features in the combined image.

FIG. 4 schematically illustrates an example of how processor 102determines, in step 304 above, which of the plurality of images providea combined image of the object showing the most detail of the object.

In this example, processor 102 receives first and second images from theplurality of different-illumination-pattern images (step 402) (e.g.,images 21-26). Processor 102 determines a measure of information ofpixels associated with the object in the first image and the secondimage and compares the information measures for each pixel in the firstimage and second image.

In one embodiment, the information measure includes local contrastvalues for pixels associated with the object in the first image and thesecond image. In this example, processor 102, determines local contrastvalues of pixels associated with the object in the first image and thesecond image (step 404) and compares the local contrast values for eachpixel in the first image and second image, e.g., to determine a pixelwith the higher local contrast value. If a pixel from the object in thefirst image has a local contrast value higher than the same pixel in thesecond image (decision point 406) then the pixel from the first image isused (step 407) to create the combined image (step 410). If the pixelfrom the first image has a local contrast value lower than that samepixel in the second image (decision point 406) then the pixel from thesecond image is used (step 408) to create the combined image (step 410).This process can be repeated for all pixels associated with the object.As described above, pixels associated with the object may be determinedby using image analysis algorithms such as segmentation.

In some embodiments, combining images to obtain a combined imageincludes creating pixels of the combined image based on a statistic ofvalues of corresponding pixels in the first and second images.Typically, statistics (one or a combination of statistics) that providea measure of information are used, such as, local contrast value,minimum or median channel values for pixels in the first and secondimages, etc. In some embodiments, the values of pixels of the combinedimage may be based on a weighted average of values of the correspondingpixels in the first and second images. For example, a specific pixel orarea (which includes a plurality of pixels) in a first image has a pixelvalue I₁ and an information measure V₁. That corresponding pixel or areain a second image has a pixel value of I₂ and an information measure V₂,and that corresponding pixel or area in a third image has a pixel valueI₃ and an information measure V₃. The contribution of each of the threeimages to the corresponding pixel or area in the combined image ispartial and dependent on the information measure of the pixel in each ofthe images used to create the combined image. Thus, the combined image(Ic) can be a weighted average of the three images as follows:

Ic=(I ₁ ×V ₁ +I ₂ ×V ₂ +I ₃ ×V ₃)/(V ₁ +V ₂ +V ₃)

Other measurements of pixels information may be used and otherstatistical calculations may be used to provide a combined image.

In some embodiments, examples of which are schematically illustrated inFIGS. 5A and 5B, the patterns of illumination and the number ofdifferent-illumination-pattern images that can be combined to providethe most details of the object, are determined during the setup stageand are then automatically used during the inspection stage, to obtain acombined substantially glare-free image of the object for visualinspection of the object.

In one embodiment, which is exemplified in FIG. 5A, processor 102receives a plurality of different-illumination-pattern reference imagesof a first object (step 51), during the setup stage. The processordetermines, during the setup stage, which reference images have thespecific illumination pattern such that, when combined, provide acombined image of the object showing the most detail of the object (step53). This may be done, for example, by detecting pixels having thehigher local contrast value, as described above.

Then, in the inspection stage, processor 102 obtains (step 54) the samenumber of images of an inspected object (which is a second, same-typeobject), having the specific illumination patterns determined during thesetup stage, in step 53 and combines the images of the second, same-typeobject, to provide a combined image for visual inspection of the second,same-type object (step 56). This may be done, for example, by using thesame pixels or areas of the image detected (based on their informationmeasure) in the setup images of the first object, to obtain a combinedimage of the second object for visual inspection of the second object.

In some embodiments items on the inspection line may be positioned ororiented in rotation relative to each other. For example, an inspecteditem (such as item 4 in FIG. 1B) may be placed on the inspection line ata 90° or other angle relative to the items in the reference images(e.g., items 2 and/or 2′ in FIG. 1A). In this case, a correspondingrotation of the illumination pattern can be calculated and performedduring the inspection stage. For example, using light source 105illustrated in FIG. 2A, an illumination pattern determined (during thesetup stage) to provide images of item 2 that when combined will providea substantially glare-free image, includes a first illumination phasewhere all segments of the light source illuminate, a second illuminationphase where only left side segments illuminate and a third illuminationphase where only right side segments illuminate. During the inspectionstage item 4 is determined to be rotated 90° to the right, relative tothe position and orientation of item 2. In this case the illuminationpattern used on item 4 will include a first illumination phase where allsegments of the light source illuminate, a second illumination phasewhere only the uppermost segments illuminate and a third illuminationphase where only lower most segments illuminate.

Thus, during the setup stage, a processor analyzing the referenceimages, determines a number of different illumination pattern setupimages of a first object, which, when combined, provide a substantiallyglare-free image of the first object. During the inspection stage, thesame number of images of a second same-type object, with illuminationpatterns based on the illumination patterns determined in the setupstage, are obtained and combined to provide a combined image for visualinspection of the second object.

Comparing information measures of pixels between images captured atdifferent illumination/exposure patterns enables finding (during thesetup stage) the minimal number of different images required to obtain acombined image that has the most information of the object.

A minimal number of images required to obtain a substantially glare-freeimage of an object, can be determined (during the setup stage) by usingdifferent optimization techniques and algorithms. In one example, aminimal number of images is calculated using a penalty score utilizingpenalty factors which may be natural numbers (including zero). Forexample, a first penalty factor may be assigned to the number of imagesused to create the combined image, whereas a second penalty factor(which may be the same or different than the first penalty factor,typically a number having a negative sign relative to the first penaltyfactor) may be assigned to each pixel having a higher information valuein the combined image. A minimal number of images may be determined bycalculating the penalty score for all permutation groups of the imagesand/or by using optimization techniques and algorithms. Typically, thefirst penalty factor will attempt to impose a small as possible numberof images used to create the combined image, while the second penaltyfactor will attempt to impose a combined image showing the most detailsof the imaged object.

This process enables determining, in the setup stage, a minimal numberof images required to obtain a substantially glare-free image of anobject. This minimal number of images is then be used during theinspection stage. Obtaining the least possible number of images of eachitem during inspection, enables a quick and therefore efficientinspection process.

In some embodiments the first object (imaged during the setup stage) isa defect-free object and the second, same-type object (imaged during theinspection stage), is either defect-free or defected.

In the example schematically illustrated in FIG. 5B, processor 102controls illumination from a light source to illuminate a set of pulsesof light which includes several patterns (step 501) to enable the camerato capture a plurality of different-illumination-pattern images in thesetup stage. The processor determines (step 503), as described above,during the setup stage, which subset of illumination patterns (some ofthe illumination patterns used in step 501, preferably, a minimalsubset) enable a combined image of the object showing the most detail ofthe object.

In the inspection stage, processor 102 controls the light source (step504) to illuminate the subset of illumination patterns determined instep 503, and combines the images obtained while illuminating, toprovide a combined image for visual inspection of the second, same-typeobject (step 506).

As schematically illustrated in FIG. 6 , embodiments of the inventioncan include obtaining, typically at the setup stage, an initial image ofthe object on the inspection line (step 602), the initial image obtainedin a first specific illumination pattern, typically a pattern includingillumination of all segments of the light source (such as image 22). Theinitial image is obtained prior to obtaining the plurality ofdifferent-illumination-pattern images, to check if the initial imageincludes areas of high reflectance or glare.

Determining if an image includes areas of high reflectance can be donebased on user input and/or by applying image processing algorithms onthe image to detect, for example a number (e.g., above a threshold) ofclosely related saturated pixels. For example, if, while illuminating acamera FOV, there is a burst in saturation level in a pixel or adjacentpixels, this can indicate that using this illumination pattern willresult in saturated areas on the object. In another example, if there isa pixel or adjacent pixels for which there is a drop in the informationlevel while the illumination intensity rises, this can indicate glare orsaturation.

If the initial image does not include areas of high reflectance(decision point 604) then the specific illumination pattern used toobtain the initial image is used during the inspection stage to obtainan image of a same-type object (step 605) for performing visualinspection of the same-type object (step 614). If the initial imageincludes an area of high reflectance (decision point 604) then, stillduring the setup stage, a plurality of different-illumination-patternimages of the object are obtained (step 606) and processed, e.g., asdescribed herein, to determine (step 608) the illumination patterns thatproduce a subset (typically, a minimal subset) of the plurality ofimages, the subset including images that when combined, provide acombined image of the object showing the most detail of the object. Thenthe specific illumination patterns determined in step 608 are usedduring the inspection stage to obtain images of a same-type object (step610). The images are combined to provide a combined image (step 612) forperforming visual inspection of the same-type object (step 614).

In some embodiments, a high definition resolution (HDR) image of theobject is obtained for improved image quality. An HDR image can beobtained by capturing a plurality of images of an object, each imagehaving a different exposure value. Pixel values of the obtained imagesare compared to the dynamic range of the camera used to capture theseimages. A minimal number of optimal images can be determined based onthe comparison. These optimal images are then combined to obtain an HDRimage of the object.

An optimal image may be determined based on a difference between valuesof pixels of the image to a middle value of the dynamic range. Forexample, an optimal image may be an image having evenly distributedpixel values, an image with no over and/or under exposed areas (or withthe least over and/or under exposed areas), etc.

For creating an HDR image using two (or more) images, a set of two (ormore) optimal images may include images that together have pixels thatare closest to the middle of the dynamic range and/or fulfill otherconditions, as described above. For example, an HDR image can be createdby taking from two (or more) images the pixel whose value is closer tothe middle of the dynamic range of the camera.

Typically, the minimal number of optimal images and the exposure levelsat which they were captured, are both determined during the setup stage.During the inspection stage the same number of images at the sameexposure levels as determined during the setup stage, are used to obtainan HDR image of an object for visual inspection. In one embodiment anHDR image of the object on the inspection line is obtained (prior toobtaining a plurality of different-illumination-pattern images of theobject) and if the HDR image includes an area of high reflectance then aplurality of different-illumination-pattern images of the object areobtained as described herein.

In one embodiment, the plurality of different-illumination-patternimages may also serve as the plurality of images from which an HDR imagecan be constructed. Thus, during a setup stage, processor 102 maycontrol light source 105 to illuminate in several illumination patternsand at the same time control the camera 103 to capture images atdifferent exposure levels. Processor 102 can then process thedifferent-illumination-pattern images obtained at different exposurelevels to determine which exposure levels and/or which illuminationpatterns to use during the inspection stage to obtain high quality,glare-free images for visual inspection.

In the time line exemplified in FIG. 7 , methods according toembodiments of the invention enable a time effective process ofobtaining high quality images for visual inspection.

A first item (item 1) is placed on an inspection line, typically duringthe inspection stage. Item 1 is then illuminated in an illuminationpattern P1 which enables detecting the item (e.g., an illuminationpattern in which all segments of an illumination device are on), aspre-determined in the setup stage. An image is captured in theseillumination conditions and at a first exposure level E1 (which wasdetermined during the setup stage).

The image captured at E1 and in illumination pattern P1 can be used todetect the item within the image, while movements and vibrations of thecamera and/or item (e.g., due to conveyor belt and/or other machinerymovement), stop. Additionally, the camera can transition to exposurelevel E2 while the movements and vibrations of the camera and/or itemstop. The light source is controlled to illuminate at a secondillumination pattern P2 (e.g., an illumination pattern in which onlyleft hand segments or uppermost segments of an illumination device areon) and an image of the item is captured at exposure level E2 andillumination pattern P2. While the camera parameters are set at exposurelevel E2, the illumination device is controlled to illuminate atanother, different, illumination pattern P3 (e.g., an illuminationpattern in which only right hand segments or lowermost segments of anillumination device are on). The processor then combines the imagecaptured at E1 and illumination pattern P1 and the two images capturedat exposure level E2 and illumination patterns P2 and P3, providing asubstantially glare-free HDR image.

Since the object has already been detected, the processor mayimmediately use the glare-free HDR image for visual inspection (e.g.,displaying the image and/or running visual inspection algorithms on theimage). In parallel, the camera transitions back to exposure level E1and to illumination pattern P1 to obtain an image of a next object (item2). Thus, in some embodiments processor 102 controls transitioning froma first illumination pattern to a second illumination pattern whileapplying visual inspection algorithms on the combined glare-free image.

Item 2 is detected in the image(s) obtained at exposure level E1 andillumination pattern P1. As discussed above, item 2 may be orientateddifferently than the same type item used during the set up process,while the exposure levels and illumination patterns are determined. Ifthere is a change of orientation, it is detected in the image(s)captured at exposure level E1. Once a change in orientation is detected,the second illumination pattern used during exposure level E2 is rotatedaccording to the detected change in orientation of the item, such thatthe second illumination pattern P2′ is rotated compared to illuminationpattern P2. Similarly, the third illumination P3′ pattern used to imageitem 2, is rotated compared to illumination pattern P3. For example, foran illumination device including a square divided to equal segments, ifin illumination pattern P2 only the left hand segments of theillumination device are on, for an item 2 that is in an orientationperpendicular compared to item 1 (and compared to the orientation of theitem imaged during the set up stage) illumination pattern P2′ willinclude only the uppermost segments being on. Similarly, if illuminationpattern P3 includes only the right hand segments of the illuminationdevice being on, illumination pattern P3′ will include only thelowermost segments being on.

The embodiments described herein enable an automated visual inspectionprocess, requiring minimal user involvement, which provides highquality, unobscured images of objects, for visual inspection in anefficient inspection process.

1. A method for an automated visual inspection process of an object onan inspection line, the method comprising: illuminating the object onthe inspection line, with pulses of a first pattern illumination andpulses of a second pattern illumination, the pulses being repeated at aconstant frequency, to provide a flicker-free inspection environment forhuman workers, wherein the pulses of the first pattern and secondpattern are temporally offset from each other; controlling cameraexposure events in synchronization with the pulses to capture aplurality of images, each of the images illuminated with a differentillumination pattern; and using at least one of the plurality of imagesto perform visual inspection of the object.
 2. (canceled)
 3. The methodof claim 12 comprising: obtaining an image at a time that coincides withthe overlap between a pulse of the first pattern and a pulse of thesecond pattern; and using the image to perform visual inspection of theobject.
 4. The method of claim 18 comprising: combining the imageobtained during the overlap between the pulse of the first pattern andthe pulse of the second pattern with the first image or the secondimage, to obtain a combined image; and using the combined image topreform visual inspection of the object.
 5. The method of claim 19comprising displaying the combined image to a user.
 6. The method ofclaim 19 comprising creating pixels of the combined image based on astatistic of values of corresponding pixels in each of the first andsecond images.
 7. The method of claim 6 wherein values of pixels of thecombined image are based on a weighted average of values of thecorresponding pixels in each of the first and second images.
 8. Themethod of claim 1 wherein the automated visual inspection processcomprises a setup stage prior to an inspection stage in which an objectis inspected, the method comprising: during the setup stage, determininga minimal number of images and their corresponding illuminationpatterns, such that, when combined, provide a combined image of a firstobject showing maximal detail of the first object; during the inspectionstage, obtaining the minimal number of images of a second, same-typeobject, having the corresponding illumination patterns determined duringthe setup stage; and combining the images of the second, same-typeobject, to provide a combined image to preform visual inspection of thesecond, same-type object.
 9. The method of claim 8 wherein the firstobject is a defect-free object and wherein the second, same-type object,is either defect-free or defected.
 10. The method of claim 8 comprisingobtaining, during the inspection stage, the minimal number of images ofthe second object, having illumination patterns that are based on anorientation of the second object in relation to an orientation of thefirst object.
 11. The method of claim 1 wherein the object comprises anarea of interest within an item.
 12. The method of claim 1 comprisingobtaining a high dynamic range (HDR) image of the object on theinspection line prior to obtaining the plurality of images of theobject; and if the HDR image comprises an area of high reflectance, thenobtaining the plurality of images of the object.
 13. A system forautomated visual inspection, the system comprising a camera configuredto capture images of an object on an inspection line; a light source toproduce light pulses repeated at a constant frequency, the light pulsestemporally offset, and illuminating a field of view (FOV) of the camerain a first pattern of illumination and second pattern of illumination;and a processor in communication with the camera and light source, theprocessor to control the light source to differentially illuminate theFOV of the camera and to synchronize exposure events of the camera withthe light pulses such that exposure events of the camera capture a firstpattern of illumination and second pattern of illumination separatelyand simultaneously, to produce different illumination pattern images ina flicker-free inspection environment for human workers.
 14. (canceled)15. The system of claim 13 wherein the processor is to combine imagesobtained by the camera to provide a combined image; and to use thecombined image to perform visual inspection of the object.
 16. Thesystem of claim 13 wherein the processor is to control the light sourceto differentially illuminate the FOV of the camera based on anorientation of the object within the images.
 17. The method of claim 1wherein the pulses of the first pattern and second pattern partiallyoverlap in time, such that exposure events of the camera capture thefirst and second illumination patterns separately and simultaneously.18. The method of claim 17 comprising obtaining a first image at a timethat coincides with a pulse from the first pattern but not a pulse fromthe second pattern; obtaining a second image at a time that coincideswith a pulse from the second pattern but not a pulse from the firstpattern; and using at least one of the first and second images toperform visual inspection of the object.
 19. The method of claim 1comprising: combining at least a first image and second image from theplurality of images to provide a combined image; and using the combinedimage to perform visual inspection of the object.